{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "socSJe925zFv"
   },
   "source": [
    "#  Working With Datasets\n",
    "\n",
    "Data is central to machine learning.  This tutorial introduces the `Dataset` class that DeepChem uses to store and manage data.  It provides simple but powerful tools for efficiently working with large amounts of data.  It also is designed to easily interact with other popular Python frameworks such as NumPy, Pandas, TensorFlow, and PyTorch.\n",
    "\n",
    "## Colab\n",
    "\n",
    "This tutorial and the rest in this sequence can be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/Working_With_Datasets.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "CMWAv-Z46nCc",
    "outputId": "9ae7cfd0-ebbf-40b0-f6f1-2940cf32a839"
   },
   "outputs": [],
   "source": [
    "!pip install --pre deepchem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jk47QTZ95zF-"
   },
   "source": [
    "We can now import the `deepchem` package to play with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "PDiY03h35zF_",
    "outputId": "cdd7401d-19a0-4476-9297-b04defc67178"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.4.0-rc1.dev'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import deepchem as dc\n",
    "dc.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B0u7qIZd5zGG"
   },
   "source": [
    "# Anatomy of a Dataset\n",
    "\n",
    "In the last tutorial we loaded the Delaney dataset of molecular solubilities.  Let's load it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "saTaOpXY5zGI"
   },
   "outputs": [],
   "source": [
    "tasks, datasets, transformers = dc.molnet.load_delaney(featurizer='GraphConv')\n",
    "train_dataset, valid_dataset, test_dataset = datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F922OPtL5zGM"
   },
   "source": [
    "We now have three Dataset objects: the training, validation, and test sets.  What information does each of them contain?  We can start to get an idea by printing out the string representation of one of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "YEDcUsz35zGO",
    "outputId": "5a05747f-8b06-407d-9b11-790a1b4d1c8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<DiskDataset X.shape: (113,), y.shape: (113, 1), w.shape: (113, 1), ids: ['C1c2ccccc2c3ccc4ccccc4c13' 'COc1ccccc1Cl'\n",
      " 'COP(=S)(OC)Oc1cc(Cl)c(Br)cc1Cl' ... 'CCSCCSP(=S)(OC)OC' 'CCC(C)C'\n",
      " 'COP(=O)(OC)OC(=CCl)c1cc(Cl)c(Cl)cc1Cl'], task_names: ['measured log solubility in mols per litre']>\n"
     ]
    }
   ],
   "source": [
    "print(test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E8UCFrrN5zGf"
   },
   "source": [
    "There's a lot of information there, so let's start at the beginning.  It begins with the label \"DiskDataset\".  Dataset is an abstract class.  It has a few subclasses that correspond to different ways of storing data.\n",
    "\n",
    "- `DiskDataset` is a dataset that has been saved to disk.  The data is stored in a way that can be efficiently accessed, even if the total amount of data is far larger than your computer's memory.\n",
    "- `NumpyDataset` is an in-memory dataset that holds all the data in NumPy arrays.  It is a useful tool when manipulating small to medium sized datasets that can fit entirely in memory.\n",
    "- `ImageDataset` is a more specialized class that stores some or all of the data in image files on disk.  It is useful when working with models that have images as their inputs or outputs.\n",
    "\n",
    "Now let's consider the contents of the Dataset.  Every Dataset stores a list of *samples*.  Very roughly speaking, a sample is a single data point.  In this case, each sample is a molecule.  In other datasets a sample might correspond to an experimental assay, a cell line, an image, or many other things.  For every sample the dataset stores the following information.\n",
    "\n",
    "- The *features*, referred to as `X`.  This is the input that should be fed into a model to represent the sample.\n",
    "- The *labels*, referred to as `y`.  This is the desired output from the model.  During training, it tries to make the model's output for each sample as close as possible to `y`.\n",
    "- The *weights*, referred to as `w`.  This can be used to indicate that some data values are more important than others.  In later tutorials we will see examples of how this is useful.\n",
    "- An *ID*, which is a unique identifier for the sample.  This can be anything as long as it is unique.  Sometimes it is just an integer index, but in this dataset the ID is a SMILES string describing the molecule.\n",
    "\n",
    "Notice that `X`, `y`, and `w` all have 113 as the size of their first dimension.  That means this dataset contains 113 samples.\n",
    "\n",
    "The final piece of information listed in the output is `task_names`.  Some datasets contain multiple pieces of information for each sample.  For example, if a sample represents a molecule, the dataset might record the results of several different experiments on that molecule.  This dataset has only a single task: \"measured log solubility in mols per litre\".  Also notice that `y` and `w` each have shape (113, 1).  The second dimension of these arrays usually matches the number of tasks.\n",
    "\n",
    "# Accessing Data from a Dataset\n",
    "\n",
    "There are many ways to access the data contained in a dataset.  The simplest is just to directly access the `X`, `y`, `w`, and `ids` properties.  Each of these returns the corresponding information as a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e5K3rdGV5zGg"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.7065408738415053],\n",
       "       [0.2911162036252904],\n",
       "       [-1.4272475857596547],\n",
       "       [-0.9254664241210759],\n",
       "       [-1.9526976701170347],\n",
       "       [1.3514839414275706],\n",
       "       [-0.8591934405084332],\n",
       "       [-0.6509069205829855],\n",
       "       [-0.32900957160729316],\n",
       "       [0.6082797680572224],\n",
       "       [1.8295961803473488],\n",
       "       [1.6213096604219008],\n",
       "       [1.3751528641463715],\n",
       "       [0.45632528420252055],\n",
       "       [1.0532555151706793],\n",
       "       [-1.1053502367839627],\n",
       "       [-0.2011973889257683],\n",
       "       [0.3479216181504126],\n",
       "       [-0.9870056231899582],\n",
       "       [-0.8161160011602158],\n",
       "       [0.8402352107014712],\n",
       "       [0.22815686919328],\n",
       "       [0.06247441016167367],\n",
       "       [1.040947675356903],\n",
       "       [-0.5197810887208284],\n",
       "       [0.8023649343513898],\n",
       "       [-0.41895147793873655],\n",
       "       [-2.5964923680684198],\n",
       "       [1.7443880585596654],\n",
       "       [0.45206487811313645],\n",
       "       [0.233837410645792],\n",
       "       [-1.7917489956291888],\n",
       "       [0.7739622270888287],\n",
       "       [1.0011838851893173],\n",
       "       [-0.05445006806920272],\n",
       "       [1.1043803882432892],\n",
       "       [0.7597608734575482],\n",
       "       [-0.7001382798380905],\n",
       "       [0.8213000725264304],\n",
       "       [-1.3136367567094103],\n",
       "       [0.4567986626568967],\n",
       "       [-0.5732728540653187],\n",
       "       [0.4094608172192949],\n",
       "       [-0.3242757870635329],\n",
       "       [-0.049716283525442634],\n",
       "       [-0.39054877067617544],\n",
       "       [-0.08095926151425996],\n",
       "       [-0.2627365879946506],\n",
       "       [-0.5467636606202616],\n",
       "       [1.997172153196459],\n",
       "       [-0.03551492989416198],\n",
       "       [1.4508934168465344],\n",
       "       [-0.8639272250521937],\n",
       "       [0.23904457364392848],\n",
       "       [0.5278054308132993],\n",
       "       [-0.48475108309700315],\n",
       "       [0.2248432200126478],\n",
       "       [0.3431878336066523],\n",
       "       [1.5029650468278963],\n",
       "       [-0.4946920306388995],\n",
       "       [0.3479216181504126],\n",
       "       [0.7928973652638694],\n",
       "       [0.5609419226196206],\n",
       "       [-0.13965818985688602],\n",
       "       [-0.13965818985688602],\n",
       "       [0.15857023640000523],\n",
       "       [1.6071083067906202],\n",
       "       [1.9006029485037514],\n",
       "       [-0.7171799041956278],\n",
       "       [-0.8165893796145915],\n",
       "       [-0.13019062076936566],\n",
       "       [-0.24380144981960986],\n",
       "       [-0.14912575894440638],\n",
       "       [0.9538460397517154],\n",
       "       [-0.07811899078800374],\n",
       "       [-0.18226225075072758],\n",
       "       [0.2532459272752089],\n",
       "       [0.6887541053011454],\n",
       "       [0.044012650441008896],\n",
       "       [-0.5514974451640217],\n",
       "       [-0.2580028034508905],\n",
       "       [-0.021313576262881533],\n",
       "       [-2.4128215277705247],\n",
       "       [0.07336211461232214],\n",
       "       [0.9017744097703536],\n",
       "       [1.9384732248538328],\n",
       "       [0.8402352107014712],\n",
       "       [-0.10652169805056463],\n",
       "       [1.07692443788948],\n",
       "       [-0.403803367398704],\n",
       "       [1.2662758196398873],\n",
       "       [-0.2532690189071302],\n",
       "       [0.29064282517091444],\n",
       "       [0.9443784706641951],\n",
       "       [-0.41563782875810434],\n",
       "       [-0.7370617992794205],\n",
       "       [-1.0012069768212388],\n",
       "       [0.46626623174441706],\n",
       "       [0.3758509469585975],\n",
       "       [-0.46628932337633816],\n",
       "       [1.2662758196398873],\n",
       "       [-1.4968342185529295],\n",
       "       [-0.17800184466134344],\n",
       "       [0.8828392715953128],\n",
       "       [-0.6083028596891439],\n",
       "       [-2.170451759130003],\n",
       "       [0.32898647997537184],\n",
       "       [0.3005837727128107],\n",
       "       [0.6461500444073038],\n",
       "       [1.5058053175541524],\n",
       "       [-0.007585601085977053],\n",
       "       [-0.049716283525442634],\n",
       "       [-0.6849901692980588]], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset.y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_Zcd7jTd5zGr"
   },
   "source": [
    "This is a very easy way to access data, but you should be very careful about using it.  This requires the data for all samples to be loaded into memory at once.  That's fine for small datasets like this one, but for large datasets it could easily take more memory than you have.\n",
    "\n",
    "A better approach is to iterate over the dataset.  That lets it load just a little data at a time, process it, then free the memory before loading the next bit.  You can use the `itersamples()` method to iterate over samples one at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "LJc90fs_5zGs",
    "outputId": "8c9fd5ab-e23a-40dc-9292-8b4ff3a86890"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.70654087] C1c2ccccc2c3ccc4ccccc4c13\n",
      "[0.2911162] COc1ccccc1Cl\n",
      "[-1.42724759] COP(=S)(OC)Oc1cc(Cl)c(Br)cc1Cl\n",
      "[-0.92546642] ClC(Cl)CC(=O)NC2=C(Cl)C(=O)c1ccccc1C2=O\n",
      "[-1.95269767] ClC(Cl)C(c1ccc(Cl)cc1)c2ccc(Cl)cc2 \n",
      "[1.35148394] COC(=O)C=C\n",
      "[-0.85919344] CN(C)C(=O)Nc2ccc(Oc1ccc(Cl)cc1)cc2\n",
      "[-0.65090692] N(=Nc1ccccc1)c2ccccc2\n",
      "[-0.32900957] CC(C)c1ccc(C)cc1\n",
      "[0.60827977] Oc1c(Cl)cccc1Cl\n",
      "[1.82959618] OCC2OC(OC1(CO)OC(CO)C(O)C1O)C(O)C(O)C2O \n",
      "[1.62130966] OC1C(O)C(O)C(O)C(O)C1O\n",
      "[1.37515286] Cn2c(=O)n(C)c1ncn(CC(O)CO)c1c2=O\n",
      "[0.45632528] OCC(NC(=O)C(Cl)Cl)C(O)c1ccc(cc1)N(=O)=O\n",
      "[1.05325552] CCC(O)(CC)CC\n",
      "[-1.10535024] CC45CCC2C(CCC3CC1SC1CC23C)C4CCC5O\n",
      "[-0.20119739] Brc1ccccc1Br\n",
      "[0.34792162] Oc1c(Cl)cc(Cl)cc1Cl\n",
      "[-0.98700562] CCCN(CCC)c1c(cc(cc1N(=O)=O)S(N)(=O)=O)N(=O)=O\n",
      "[-0.816116] C2c1ccccc1N(CCF)C(=O)c3ccccc23 \n",
      "[0.84023521] CC(C)C(=O)C(C)C\n",
      "[0.22815687] O=C1NC(=O)NC(=O)C1(C(C)C)CC=C(C)C\n",
      "[0.06247441] c1c(O)C2C(=O)C3cc(O)ccC3OC2cc1(OC)\n",
      "[1.04094768] Cn1cnc2n(C)c(=O)n(C)c(=O)c12\n",
      "[-0.51978109] CC(=O)SC4CC1=CC(=O)CCC1(C)C5CCC2(C)C(CCC23CCC(=O)O3)C45\n",
      "[0.80236493] Cc1ccc(O)cc1C\n",
      "[-0.41895148] O(c1ccccc1)c2ccccc2\n",
      "[-2.59649237] Clc1cc(Cl)c(cc1Cl)c2cc(Cl)c(Cl)cc2Cl \n",
      "[1.74438806] NC(=O)c1cccnc1 \n",
      "[0.45206488] Sc1ccccc1\n",
      "[0.23383741] CNC(=O)Oc1cc(C)cc(C)c1\n",
      "[-1.791749] ClC1CC2C(C1Cl)C3(Cl)C(=C(Cl)C2(Cl)C3(Cl)Cl)Cl\n",
      "[0.77396223] CSSC\n",
      "[1.00118389] NC(=O)c1ccccc1\n",
      "[-0.05445007] Clc1ccccc1Br\n",
      "[1.10438039] COC(=O)c1ccccc1OC2OC(COC3OCC(O)C(O)C3O)C(O)C(O)C2O\n",
      "[0.75976087] CCCCC(O)CC\n",
      "[-0.70013828] CCN2c1nc(C)cc(C)c1NC(=O)c3cccnc23 \n",
      "[0.82130007] Oc1cc(Cl)cc(Cl)c1\n",
      "[-1.31363676] Cc1cccc2c1ccc3ccccc32\n",
      "[0.45679866] CCCCC(CC)CO\n",
      "[-0.57327285] CC(C)N(C(C)C)C(=O)SCC(=CCl)Cl\n",
      "[0.40946082] Cc1ccccc1\n",
      "[-0.32427579] Clc1cccc(n1)C(Cl)(Cl)Cl\n",
      "[-0.04971628] C1CCC=CCC1\n",
      "[-0.39054877] CN(C)C(=S)SSC(=S)N(C)C \n",
      "[-0.08095926] COC1=CC(=O)CC(C)C13Oc2c(Cl)c(OC)cc(OC)c2C3=O\n",
      "[-0.26273659] CCCCCCCCCCO\n",
      "[-0.54676366] CCC(C)(C)CC\n",
      "[1.99717215] CNC(=O)C(C)SCCSP(=O)(OC)(OC)\n",
      "[-0.03551493] Oc1cc(Cl)c(Cl)c(Cl)c1Cl\n",
      "[1.45089342] CCCC=O\n",
      "[-0.86392723] CC4CC3C2CCC1=CC(=O)C=CC1(C)C2(F)C(O)CC3(C)C4(O)C(=O)COC(C)=O \n",
      "[0.23904457] CCCC\n",
      "[0.52780543] COc1ccccc1O\n",
      "[-0.48475108] CC1CC2C3CCC(O)(C(=O)C)C3(C)CC(O)C2(F)C4(C)C=CC(=O)C=C14\n",
      "[0.22484322] ClC(Cl)C(Cl)(Cl)Cl\n",
      "[0.34318783] CCOC(=O)c1ccccc1C(=O)OCC\n",
      "[1.50296505] CC(C)CO\n",
      "[-0.49469203] CC(C)Cc1ccccc1\n",
      "[0.34792162] ICI\n",
      "[0.79289737] CCCC(O)CCC\n",
      "[0.56094192] CCCCCOC(=O)C\n",
      "[-0.13965819] Oc1c(Cl)c(Cl)cc(Cl)c1Cl\n",
      "[-0.13965819] CCCc1ccccc1\n",
      "[0.15857024] FC(F)(Cl)C(F)(F)Cl\n",
      "[1.60710831] CC=CC=O\n",
      "[1.90060295] CN(C)C(=O)N(C)C \n",
      "[-0.7171799] Cc1cc(C)c(C)cc1C\n",
      "[-0.81658938] CC(=O)OC3(CCC4C2CCC1=CC(=O)CCC1C2CCC34C)C#C\n",
      "[-0.13019062] CCOP(=S)(OCC)N2C(=O)c1ccccc1C2=O\n",
      "[-0.24380145] c1ccccc1NC(=O)c2c(O)cccc2\n",
      "[-0.14912576] CCN(CC)C(=S)SCC(Cl)=C\n",
      "[0.95384604] ClCC\n",
      "[-0.07811899] CC(=O)Nc1cc(NS(=O)(=O)C(F)(F)F)c(C)cc1C\n",
      "[-0.18226225] O=C(C=CC=Cc2ccc1OCOc1c2)N3CCCCC3\n",
      "[0.25324593] CC/C=C\\C\n",
      "[0.68875411] CNC(=O)ON=C(CSC)C(C)(C)C \n",
      "[0.04401265] O=C2NC(=O)C1(CCCCCCC1)C(=O)N2\n",
      "[-0.55149745] c1(C(C)(C)C)cc(C(C)(C)C)cc(OC(=O)NC)c1\n",
      "[-0.2580028] Oc2cc(O)c1C(=O)CC(Oc1c2)c3ccc(O)c(O)c3\n",
      "[-0.02131358] O=C(c1ccccc1)c2ccccc2\n",
      "[-2.41282153] CCCCCCCCCCCCCCCCCCCC\n",
      "[0.07336211] N(Nc1ccccc1)c2ccccc2 \n",
      "[0.90177441] CCC(CC)CO\n",
      "[1.93847322] Oc1ccncc1\n",
      "[0.84023521] Cl\\C=C/Cl\n",
      "[-0.1065217] CC1CCCC1\n",
      "[1.07692444] CC(C)CC(C)O\n",
      "[-0.40380337] O2c1ccc(N)cc1N(C)C(=O)c3cc(C)ccc23 \n",
      "[1.26627582] CC(C)(C)CO\n",
      "[-0.25326902] CC(C)(C)C(=O)C(Oc1ccc(Cl)cc1)n2cncn2\n",
      "[0.29064283] Cc1cc(no1)C(=O)NNCc2ccccc2\n",
      "[0.94437847] CC=C\n",
      "[-0.41563783] Oc1ccc(Cl)cc1Cc2cc(Cl)ccc2O\n",
      "[-0.7370618] CCOC(=O)Nc2cccc(OC(=O)Nc1ccccc1)c2 \n",
      "[-1.00120698] O=C1c2ccccc2C(=O)c3ccccc13\n",
      "[0.46626623] CCCCCCC(C)O\n",
      "[0.37585095] CC1=C(C(=O)Nc2ccccc2)S(=O)(=O)CCO1\n",
      "[-0.46628932] CCCCc1ccccc1\n",
      "[1.26627582] O=C1NC(=O)C(=O)N1 \n",
      "[-1.49683422] COP(=S)(OC)Oc1ccc(Sc2ccc(OP(=S)(OC)OC)cc2)cc1\n",
      "[-0.17800184] NS(=O)(=O)c1cc(ccc1Cl)C2(O)NC(=O)c3ccccc23\n",
      "[0.88283927] CC(C)COC(=O)C\n",
      "[-0.60830286] CC(C)C(C)(C)C\n",
      "[-2.17045176] Clc1ccc(c(Cl)c1Cl)c2c(Cl)cc(Cl)c(Cl)c2Cl \n",
      "[0.32898648] N#Cc1ccccc1C#N\n",
      "[0.30058377] Cc1cccc(c1)N(=O)=O\n",
      "[0.64615004] FC(F)(F)C(Cl)Br \n",
      "[1.50580532] CNC(=O)ON=C(SC)C(=O)N(C)C\n",
      "[-0.0075856] CCSCCSP(=S)(OC)OC\n",
      "[-0.04971628] CCC(C)C\n",
      "[-0.68499017] COP(=O)(OC)OC(=CCl)c1cc(Cl)c(Cl)cc1Cl\n"
     ]
    }
   ],
   "source": [
    "for X, y, w, id in test_dataset.itersamples():\n",
    "    print(y, id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aQa88cbj5zGw"
   },
   "source": [
    "Most deep learning models can process a batch of multiple samples all at once.  You can use `iterbatches()` to iterate over batches of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "HSVqeYox5zGx",
    "outputId": "270a6a17-6238-4081-b0cf-3f17e23f4bb5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50, 1)\n",
      "(50, 1)\n",
      "(13, 1)\n"
     ]
    }
   ],
   "source": [
    "for X, y, w, ids in test_dataset.iterbatches(batch_size=50):\n",
    "    print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`iterbatches()` has other features that are useful when training models.  For example, `iterbatches(batch_size=100, epochs=10, deterministic=False)` will iterate over the complete dataset ten times, each time with the samples in a different random order.\n",
    "\n",
    "Datasets can also expose data using the standard interfaces for TensorFlow and PyTorch.  To get a `tensorflow.data.Dataset`, call `make_tf_dataset()`.  To get a `torch.utils.data.IterableDataset`, call `make_pytorch_dataset()`.  See the API documentation for more details.\n",
    "\n",
    "The final way of accessing data is `to_dataframe()`.  This copies the data into a Pandas `DataFrame`.  This requires storing all the data in memory at once, so you should only use it with small datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>y</th>\n",
       "      <th>w</th>\n",
       "      <th>ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-1.706541</td>\n",
       "      <td>1.0</td>\n",
       "      <td>C1c2ccccc2c3ccc4ccccc4c13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>0.291116</td>\n",
       "      <td>1.0</td>\n",
       "      <td>COc1ccccc1Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-1.427248</td>\n",
       "      <td>1.0</td>\n",
       "      <td>COP(=S)(OC)Oc1cc(Cl)c(Br)cc1Cl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-0.925466</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ClC(Cl)CC(=O)NC2=C(Cl)C(=O)c1ccccc1C2=O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-1.952698</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ClC(Cl)C(c1ccc(Cl)cc1)c2ccc(Cl)cc2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>0.646150</td>\n",
       "      <td>1.0</td>\n",
       "      <td>FC(F)(F)C(Cl)Br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>1.505805</td>\n",
       "      <td>1.0</td>\n",
       "      <td>CNC(=O)ON=C(SC)C(=O)N(C)C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-0.007586</td>\n",
       "      <td>1.0</td>\n",
       "      <td>CCSCCSP(=S)(OC)OC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-0.049716</td>\n",
       "      <td>1.0</td>\n",
       "      <td>CCC(C)C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>&lt;deepchem.feat.mol_graphs.ConvMol object at 0x...</td>\n",
       "      <td>-0.684990</td>\n",
       "      <td>1.0</td>\n",
       "      <td>COP(=O)(OC)OC(=CCl)c1cc(Cl)c(Cl)cc1Cl</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>113 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     X         y    w  \\\n",
       "0    <deepchem.feat.mol_graphs.ConvMol object at 0x... -1.706541  1.0   \n",
       "1    <deepchem.feat.mol_graphs.ConvMol object at 0x...  0.291116  1.0   \n",
       "2    <deepchem.feat.mol_graphs.ConvMol object at 0x... -1.427248  1.0   \n",
       "3    <deepchem.feat.mol_graphs.ConvMol object at 0x... -0.925466  1.0   \n",
       "4    <deepchem.feat.mol_graphs.ConvMol object at 0x... -1.952698  1.0   \n",
       "..                                                 ...       ...  ...   \n",
       "108  <deepchem.feat.mol_graphs.ConvMol object at 0x...  0.646150  1.0   \n",
       "109  <deepchem.feat.mol_graphs.ConvMol object at 0x...  1.505805  1.0   \n",
       "110  <deepchem.feat.mol_graphs.ConvMol object at 0x... -0.007586  1.0   \n",
       "111  <deepchem.feat.mol_graphs.ConvMol object at 0x... -0.049716  1.0   \n",
       "112  <deepchem.feat.mol_graphs.ConvMol object at 0x... -0.684990  1.0   \n",
       "\n",
       "                                         ids  \n",
       "0                  C1c2ccccc2c3ccc4ccccc4c13  \n",
       "1                               COc1ccccc1Cl  \n",
       "2             COP(=S)(OC)Oc1cc(Cl)c(Br)cc1Cl  \n",
       "3    ClC(Cl)CC(=O)NC2=C(Cl)C(=O)c1ccccc1C2=O  \n",
       "4        ClC(Cl)C(c1ccc(Cl)cc1)c2ccc(Cl)cc2   \n",
       "..                                       ...  \n",
       "108                         FC(F)(F)C(Cl)Br   \n",
       "109                CNC(=O)ON=C(SC)C(=O)N(C)C  \n",
       "110                        CCSCCSP(=S)(OC)OC  \n",
       "111                                  CCC(C)C  \n",
       "112    COP(=O)(OC)OC(=CCl)c1cc(Cl)c(Cl)cc1Cl  \n",
       "\n",
       "[113 rows x 4 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset.to_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Datasets\n",
    "\n",
    "Now let's talk about how you can create your own datasets.  Creating a `NumpyDataset` is very simple: just pass the arrays containing the data to the constructor.  Let's create some random arrays, then wrap them in a NumpyDataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NumpyDataset X.shape: (10, 5), y.shape: (10, 2), w.shape: (10, 1), ids: [0 1 2 3 4 5 6 7 8 9], task_names: [0 1]>\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "X = np.random.random((10, 5))\n",
    "y = np.random.random((10, 2))\n",
    "dataset = dc.data.NumpyDataset(X=X, y=y)\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we did not specify weights or IDs.  These are optional, as is `y` for that matter.  Only `X` is required.  Since we left them out, it automatically built `w` and `ids` arrays for us, setting all weights to 1 and setting the IDs to integer indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>y1</th>\n",
       "      <th>y2</th>\n",
       "      <th>w</th>\n",
       "      <th>ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.547330</td>\n",
       "      <td>0.919941</td>\n",
       "      <td>0.289138</td>\n",
       "      <td>0.431806</td>\n",
       "      <td>0.776672</td>\n",
       "      <td>0.532579</td>\n",
       "      <td>0.443258</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.980867</td>\n",
       "      <td>0.642487</td>\n",
       "      <td>0.460640</td>\n",
       "      <td>0.500153</td>\n",
       "      <td>0.014848</td>\n",
       "      <td>0.678259</td>\n",
       "      <td>0.274029</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.953254</td>\n",
       "      <td>0.704446</td>\n",
       "      <td>0.857458</td>\n",
       "      <td>0.378372</td>\n",
       "      <td>0.705789</td>\n",
       "      <td>0.704786</td>\n",
       "      <td>0.901080</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.904970</td>\n",
       "      <td>0.729710</td>\n",
       "      <td>0.304247</td>\n",
       "      <td>0.861546</td>\n",
       "      <td>0.917029</td>\n",
       "      <td>0.121747</td>\n",
       "      <td>0.758845</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.464144</td>\n",
       "      <td>0.059168</td>\n",
       "      <td>0.600405</td>\n",
       "      <td>0.880529</td>\n",
       "      <td>0.688043</td>\n",
       "      <td>0.595495</td>\n",
       "      <td>0.719861</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.820482</td>\n",
       "      <td>0.139002</td>\n",
       "      <td>0.627421</td>\n",
       "      <td>0.129399</td>\n",
       "      <td>0.920024</td>\n",
       "      <td>0.634030</td>\n",
       "      <td>0.464525</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.113727</td>\n",
       "      <td>0.551801</td>\n",
       "      <td>0.536189</td>\n",
       "      <td>0.066091</td>\n",
       "      <td>0.311320</td>\n",
       "      <td>0.699331</td>\n",
       "      <td>0.171532</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.516131</td>\n",
       "      <td>0.918903</td>\n",
       "      <td>0.429036</td>\n",
       "      <td>0.844973</td>\n",
       "      <td>0.639367</td>\n",
       "      <td>0.464089</td>\n",
       "      <td>0.337989</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.809393</td>\n",
       "      <td>0.201450</td>\n",
       "      <td>0.821420</td>\n",
       "      <td>0.841390</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.230462</td>\n",
       "      <td>0.376151</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.076750</td>\n",
       "      <td>0.389277</td>\n",
       "      <td>0.350371</td>\n",
       "      <td>0.291806</td>\n",
       "      <td>0.127522</td>\n",
       "      <td>0.544606</td>\n",
       "      <td>0.306578</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         X1        X2        X3        X4        X5        y1        y2    w  \\\n",
       "0  0.547330  0.919941  0.289138  0.431806  0.776672  0.532579  0.443258  1.0   \n",
       "1  0.980867  0.642487  0.460640  0.500153  0.014848  0.678259  0.274029  1.0   \n",
       "2  0.953254  0.704446  0.857458  0.378372  0.705789  0.704786  0.901080  1.0   \n",
       "3  0.904970  0.729710  0.304247  0.861546  0.917029  0.121747  0.758845  1.0   \n",
       "4  0.464144  0.059168  0.600405  0.880529  0.688043  0.595495  0.719861  1.0   \n",
       "5  0.820482  0.139002  0.627421  0.129399  0.920024  0.634030  0.464525  1.0   \n",
       "6  0.113727  0.551801  0.536189  0.066091  0.311320  0.699331  0.171532  1.0   \n",
       "7  0.516131  0.918903  0.429036  0.844973  0.639367  0.464089  0.337989  1.0   \n",
       "8  0.809393  0.201450  0.821420  0.841390  0.100026  0.230462  0.376151  1.0   \n",
       "9  0.076750  0.389277  0.350371  0.291806  0.127522  0.544606  0.306578  1.0   \n",
       "\n",
       "  ids  \n",
       "0   0  \n",
       "1   1  \n",
       "2   2  \n",
       "3   3  \n",
       "4   4  \n",
       "5   5  \n",
       "6   6  \n",
       "7   7  \n",
       "8   8  \n",
       "9   9  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.to_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about creating a DiskDataset?  If you have the data in NumPy arrays, you can call `DiskDataset.from_numpy()` to save it to disk.  Since this is just a tutorial, we will save it to a temporary directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<DiskDataset X.shape: (10, 5), y.shape: (10, 2), w.shape: (10, 1), ids: [0 1 2 3 4 5 6 7 8 9], task_names: [0 1]>\n"
     ]
    }
   ],
   "source": [
    "import tempfile\n",
    "\n",
    "with tempfile.TemporaryDirectory() as data_dir:\n",
    "    disk_dataset = dc.data.DiskDataset.from_numpy(X=X, y=y, data_dir=data_dir)\n",
    "    print(disk_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about larger datasets that can't fit in memory?  What if you have some huge files on disk containing data on hundreds of millions of molecules?  The process for creating a DiskDataset from them is slightly more involved.  Fortunately, DeepChem's `DataLoader` framework can automate most of the work for you.  That is a larger subject, so we will return to it in a later tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MhZxVoVs5zMa"
   },
   "source": [
    "# Congratulations! Time to join the Community!\n",
    "\n",
    "Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:\n",
    "\n",
    "## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)\n",
    "This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.\n",
    "\n",
    "## Join the DeepChem Gitter\n",
    "The DeepChem [Gitter](https://gitter.im/deepchem/Lobby) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "01_The_Basic_Tools_of_the_Deep_Life_Sciences.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
